Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: YakYak Date: Nov 9, 2007 1:58am
Forum: web Subject: Re: robots.txt only gives temporary removal of pages?

Is there a specific spider name that the Archive uses, there must be. One of our sites has databases that are not unique to the site that a few other sites use.

Some of these databases are big as 100,000 pages deep and you wont want to list these.

I think we could write a Disallow for a specific spider for a section of the site. Or am I wrong?

James - http://www.ctmf.edu

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffPaulForrest Date: Nov 9, 2007 11:14am
Forum: web Subject: Re: robots.txt only gives temporary removal of pages?

In fact you can exclude just our spider. We have two that you should exclude in your .txt file. They are ia_archiver and ia_archiver2
If you have any other questions, please let me know.

Reply to this post
Reply [edit]

Poster: adisonedu Date: Nov 29, 2007 9:58am
Forum: web Subject: Re: robots.txt only gives temporary removal of pages?

Can you give us an example. I am a little dumb when it comes to creating code. Also other people might find it easier to just cut and paste the code right into their access file.

Preston
Admin
http://www.adison.edu

This post was modified by adisonedu on 2007-11-29 17:58:13

Reply to this post
Reply [edit]

Poster: Telephone Toughguy Date: Nov 29, 2007 10:04am
Forum: web Subject: Hisorical revisionists be damned

The level at which some are trying to erase internet history are staggering and disturbing. It is quite a shame the vast loss of historical information that is lost because of these malicious text bots. Historians of the future will curse you for sure.

Reply to this post
Reply [edit]

Poster: adisonedu Date: Nov 29, 2007 10:09am
Forum: web Subject: Re: Hisorical revisionists be damned

This is not meant to erase history. To use the access file to limit the access into certain sections helps. Some of the info in these sections is not historical nor educational.

Like a link directory; why would the Archive want to store anything not of readable value. These are just links and small descriptions and hold no content value.

Preston

Reply to this post
Reply [edit]

Poster: Telephone Toughguy Date: Nov 29, 2007 10:16am
Forum: web Subject: Re: Hisorical revisionists be damned

Everything is really important to the most avid nerds in the trenches. Although nerdly, their 20 hours per day of recording every single moment will be the crucial foundations of the bibles of future civilizations, let's now leave them with 900 year old people and other partial or exaggerated text again. It's for the sake of humanity for god's sake. Come down from the mountains and free the data. Down with DRM and down with satans force of information obliteration.

Reply to this post
Reply [edit]

Poster: adisonedu Date: Nov 29, 2007 4:28pm
Forum: web Subject: Re: Hisorical revisionists be damned

"Come down from the mountains and free the data. Down with DRM and down with satans force of information obliteration."

Do you always go off the deep end? As if holding out my data is going to change the world. "satans force of information obliteration" So Satan is making me withhold this info from you?

Where do you live? In a hole.

Reply to this post
Reply [edit]

Poster: Birdland Date: Jan 5, 2008 4:40pm
Forum: web Subject: Re: Hisorical revisionists be damned

Talk about conspiracy, "information obliteration" is a mith and if you continue to go down that path you will go crazy.

The Birdman is watching you.